Benchmarks and software standards: A case study of GARCH procedures

This paper addresses the evaluation of nonlinear methods in econometric software, taking GARCH procedures as a case study. In particular, it analyzes seven widely used packages, utilizing a recently developed benchmark. Four of the packages are found to be unsuitable, in most cases because the developer either does not specifically indicate which of the many possible GARCH models is being estimated, or does not accommodate the most common model specified in the applied literature, or both. A principal finding is that implementation of the GARCH procedure varies so widely that two packages ostensibly doing the same thing actually may be estimating substantively different models. This lack of standardization raises several questions concerning the evaluation of software. These include the issues normally associated with the creation of benchmarks, but also the critical role that software plays, and can play, in the development of modern econometrics.